Multiple Systems for Value Learning

According to expected utility theory, choice is unitary by definition. For instance, a single scale mapping the objects of choice to utility or value is implicit in (indeed, formally equivalent to; see Chapter 1) a set of preferences over these objects, so long as those preferences satisfy some regularities such as transitivity. Of course, such an abstract analysis does not speak directly to the mechanisms and processes that actually produce choices. At a process level, the notion that human and animal decisions are governed not by a single unitary controller, but rather by multiple, competing sub-systems, is pervasive throughout the history of psychology (Damasio, 1994; Dickinson, 1985; Freud, 1961; James, 1950). Similar frameworks have also become prevalent in neuroscience and behavioral economics (Balleine and Dickinson 1998; Balleine et al., 2008; Daw et al., 2005; Kahneman, 2003; Laibson, 1997; Loewenstein and O’Donoghue, 2004; Thaler and An, 1981; Weber and Johnson, 2009). Although such multiplicity of control sits oddly with some theoretical perspectives, as we stress below, the brain is modular, and it evolved over time. Behavioral control mechanisms exist even in primitive organisms and are preserved, and augmented, in humans and other mammals. Also, such a multiplicity of choice mechanisms can be normatively justified even in more theoretical analyses, once computational considerations are taken into account. Although theory prescribes that a decision variable such as expected utility should take some particular value, exactly computing this value to guide choice is often laborious or intractable. In this case, as we will see, approximations may be preferable overall, and different approximations are more efficient in different circumstances. In this chapter, we focus on a particularly crisply defined and well-supported version of the multiple systems framework, which has its roots in the behavioral psychology of animal learning (Balleine and Dickinson, 1998; Dickinson, 1985), and has more recently been extended to humans and to serve as the foundation for predominant neural and computational accounts of these functions (Balleine and O’Doherty, 2010; Balleine et al., 2008; Daw et al., 2005). The overarching theme of all this work is that a particular